Without limiting the scope of the disclosed device, the background is described in connection with novel systems and methods of use for the detection obfuscation and packing in computer files and systems.
Zero-day malware detection is a persistent problem. Hundreds of thousands of new malware are produced and published on the Internet daily. Although conventional signature-based techniques are still widely relied upon, they are only useful for known malware. Many research efforts have aimed at helping flag and detect unknown suspicious and malicious files. All of these techniques can be categorized into sandbox analysis, heuristic static analysis or code emulation. Among the three, heuristic static analysis is the fastest, yet the weakest against obfuscation techniques. Code obfuscation includes packing, protecting, encrypting or inserting anti-disassembly tricks, and is used to hinder the process of reverse engineering and code analysis. About 80% to 90% of malware use some kind of packing techniques [1] and around 50% of new malware are simply packed versions of older known malware according to a 2006 article [2], and it is believed to be more than that by now. While it is very common for malware to use code obfuscation, benign executable files rarely employ such techniques. Thus, it has become a common practice to flag an obfuscated file as suspicious and then examine it with more costly analysis to determine if it is malicious or not.
Most current work of detecting obfuscated files is based on executable file structure characteristics. Many public packers, indeed, exhibit identifiable changes in the packed PE file. However, this is not always the case with custom packers and self-encrypting malware. Moreover, packing is not the only obfuscation technique used by malware writers. Malware can use anti-analysis tricks that hinder the disassembly or analysis process. Such tricks can leave absolutely no trace in the header as it is based on obfuscating the instructions sequence and the execution flow of the program. Other methods depend on detecting the signature of known packers in the file. The drawback of this method is obvious as it does not work with unknown and custom packers and cryptors. It also fails if the signature is slightly modified. Calculating the entropy score of the file is another method of identifying packed and encrypted files. This method could be effective against encryption or packing obfuscation, but is ineffective against anti-disassembly tricks. In addition, the entropy score of a file can be reduced to achieve low entropy similar to those normal program.
While all of the aforementioned systems and methods may fulfill their unique purposes, none of them fulfill the need for a practical, effective, and efficient means for the detection obfuscation and packing in computer files and systems.
Therefore, the present invention proposes a novel system and method of use for the detection obfuscation and packing in computer files and systems that addresses the shortcomings in the prior art.